Pinging @elastic/fleet (Team:Fleet)

@juliaElastic super excited to see so much progress on this already. Do we need to wait for the Fleet Server and Agent dependencies to be merged before we merge this?

@joshdover yes, we need the backend implementation of the action, otherwise it would be an action that never completes.
Also we need a real implementation of the uploads api which is blocked currently.

@elasticmachine merge upstream

ResponseOps changes look good!

We're chatting about this PR right now, since we don't think we've reviewed your usage of task manager - but can now see it when new task types get added (via the check in check_registered_task_types.ts. Thinking it would be good to get an overview of how you're using it, we may have some thoughts/hints/tips.

@pmuellr See the reasoning here: #138870
To summarize, we have Fleet bulk actions that are being run in async mode for more than 10k agents. The Kibana Task Manager is used to catch errors and trigger retry of actions in a Task, and also to check completion in a Task after 5m.
The same logic is used for all types of actions (upgrade, unenroll, reassign, update tags, request diagnostics).

💚 Build Succeeded

• Buildkite Build
• Commit: 89888ae

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

History

• 💔 Build #86339 failed 3cfaf2e
• 💔 Build #86214 failed b3123fa
• 💔 Build #86142 failed c0755cf
• 💔 Build #86131 failed a486196
• 💛 Build #85818 was flaky b164e02

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @juliaElastic

I tested the branch locally, I'm not sure if the upload part can be tested yet but up to the point works as per requirements. In my local I see that the UI gets stuck in loading state when requesting the diagnostic, so it's difficult to test the rest of the flow.

Thanks for the review!
I can document the steps how I tested e2e locally (checking out the fleet-server pr for the upload API, upload a test file with a script, adding some code to kibana that immediately saves the action result with the file id when taking the action).

I just have a question. From the agent activity is possible to see that the diagnostics where requested, but it's hard to know for which agent. It's totally fine in case of bulk requests, but would it make sense to consider adding the agent id for single requests?

We have an enhancement to add more insight into the agents actioned: #141206

I think in Request Diagnostics case this shouldn't be an issue, because when taking the action on a single agent, the UI should navigate to Agent Details / Diagnostics tab.

Here are the steps how I tested e2e locally:

• check out the fleet-server pr for the upload API: File Upload Feature fleet-server#1902
• start Fleet Server locally

make local
./bin/fleet-server --config fleet-server.dev.yml 2>&1

• upload a test file with a script: extract this zip to your fleet-server directory, run go run ./upload.go elastic-agent-diagnostics-2022-10-04T09-54-34Z-00.zip
  upload_test.zip

• add some code to kibana that immediately saves the action result with the file id when taking the action:
  replace requestDiagnostics function in x-pack/plugins/fleet/server/services/agents/request_diagnostics.ts:

export async function requestDiagnostics(
  esClient: ElasticsearchClient,
  agentId: string
): Promise<{ actionId: string }> {
  const id = new Date().toISOString();
  const response = await createAgentAction(esClient, {
    agents: [agentId],
    created_at: new Date().toISOString(),
    type: 'REQUEST_DIAGNOSTICS',
    id,
  });
  await bulkCreateAgentActionResults(
    esClient,
    [agentId].map((agent) => ({
      agentId: agent,
      actionId: id,
      data: {file_id: 'a59cdf67-65d9-4070-90ee-6b7c488215aa.b324764f-3631-40a5-8ff1-756fbbfbc3ac'} // replace with the actual file id, you can check this in console `GET .fleet-agent-files/_search`
    }))
  );
  return { actionId: response.id };
}

• add data parameter to bulkCreateAgentActionResults in actions.ts:

export async function bulkCreateAgentActionResults(
  esClient: ElasticsearchClient,
  results: Array<{
    actionId: string;
    agentId: string;
    error?: string;
    data?: any;
  }>
): Promise<void> {
  if (results.length === 0) {
    return;
  }

  const bulkBody = results.flatMap((result) => {
    const body = {
      '@timestamp': new Date().toISOString(),
      action_id: result.actionId,
      agent_id: result.agentId,
      error: result.error,
      data: result.data,
    };

    return [
      {
        create: {
          _id: uuid.v4(),
        },
      },
      body,
    ];
  });

  await esClient.bulk({
    index: AGENT_ACTIONS_RESULTS_INDEX,
    body: bulkBody,
    refresh: 'wait_for',
  });
}

• Go to Agent Details UI and trigger Request Diagnostics action. The file should immediately be available to download, and the uploaded zip file should be the same as you uploaded.

NOTE: this feature is turned off in 8.6 as the backend is not ready yet. The feature is planned to be enabled in 8.7.

Hi Team,

We have created 11 testcases for this feature under our Fleet test suite at link:

• Fleet>Agent Diagnostics

Please let us know if any other scenario is missing from our end.
Thanks